MC-KDFF: A Hybrid Deep-Learning Framework for Malware Classification Using Image Generation and Knowledge Distillation

Abstract

Cyber threats are no longer a distant concern — malware attacks are disrupting organisations, governments, and individuals on a daily basis. Yet the tools we rely on to stop them have struggled to keep pace. Traditional detection approaches, whether signature-based scanning or manual feature engineering, work reasonably well against known threats but fall apart when confronted with new, obfuscated, or polymorphic malware. This paper presents a fresh perspective on the problem: instead of analysing malware as code, we treat it as an image. By mapping the raw bytes of a malware binary to pixel values, we generate a visual representation of the file that preserves its internal structure. We then train a Convolutional Neural Network (CNN) to classify these images into known malware families. The approach bypasses the need for manual feature extraction entirely, allowing the model to learn discriminative patterns directly from the data. Our results show that this method outperforms traditional classifiers such as SVM and k-NN, generalises well to unseen samples, and scales effectively to large datasets — making it a viable candidate for real-world cybersecurity applications.

Introduction

The rapid growth of sophisticated malware has exposed the limitations of traditional signature-based detection methods, which struggle against obfuscation, polymorphism, and zero-day attacks. Static analysis is vulnerable to code obfuscation, while dynamic analysis is accurate but computationally expensive and difficult to scale. To address these challenges, this project proposes an automated malware classification system that converts malware binaries into grayscale images and uses a Convolutional Neural Network (CNN) to identify malware families based on visual patterns. By learning features directly from binary images, the approach eliminates the need for manual feature engineering and enables fast, scalable, and accurate malware detection.

A review of existing literature shows that machine learning and deep learning techniques have significantly improved malware detection compared to traditional methods. Studies have explored ensemble learning, deep neural networks, AutoML, reinforcement learning, and hybrid static-dynamic analysis for malware and ransomware detection. Although these methods achieve high accuracy, many suffer from limitations such as high computational cost, reliance on handcrafted features, poor interpretability, or difficulty handling obfuscated malware. These findings motivate the proposed image-based CNN approach, which offers better generalization, automation, and scalability.

The proposed methodology consists of collecting labeled malware binaries, converting them into grayscale images through byte-to-pixel mapping, preprocessing the images using median filtering, binarization, normalization, and data augmentation, and then training a CNN with multiple convolutional and pooling layers. The model uses ReLU activation, dropout, softmax classification, and the Adam optimizer to classify malware into ten categories. Performance is evaluated using accuracy, precision, recall, F1-score, confusion matrices, and learning curves. The implementation is developed using Python, TensorFlow, Keras, OpenCV, Flask, and MySQL, enabling real-time malware prediction through a web interface. Overall, the proposed system demonstrates an efficient, scalable, and automated solution for malware classification using deep learning and binary image analysis.

Conclusion

This project set out to answer a straightforward question: can we detect and classify malware more effectively by treating it as an image classification problem? The answer, based on everything we have built and tested, is yes — at least within the scope of the experimental setup. The core contribution is a complete, working pipeline that takes a raw malware binary, converts it to an image, preprocesses it, and feeds it into a CNN that returns a family-level classification in milliseconds. That pipeline outperforms classical baselines, generalises to unseen samples, and scales comfortably to large datasets. None of those properties are trivial to achieve simultaneously, and the image-based framing is what makes them possible together. The most practically significant finding is that automatic feature learning — letting the CNN discover what to look for rather than telling it — is not just more convenient than manual feature engineering, it is more effective. The model finds patterns in the byte structure that human analysts might never think to formalise into features. That is the genuine promise of deep learning for cybersecurity: not replacing human judgement, but augmenting it with a pattern-recognition capability that operates at a scale and speed no human can match. That said, we want to be clear about what this project does not solve. It is a static method, so it cannot catch malware that behaves differently than it looks. It needs a large, clean training dataset, which is not always available. And its CNN, like most deep neural networks, does not readily explain its decisions — a limitation that matters in contexts where an analyst needs to understand why a file was flagged, not just that it was. These are real gaps, and they point toward a natural research agenda: combining this image-based approach with dynamic features, exploring explainability techniques, and experimenting with transfer learning to reduce the data requirements. In closing, this work contributes a practical, well-evaluated, and reproducible method for malware classification that bridges cybersecurity and computer vision. It is not the last word on the subject — no single paper could be — but it is, we hope, a useful and honest step forward.

References

[1] Akhtar, Muhammad Shoaib, and Tao Feng. \"Malware analysis and detection using machine learning algorithms.\" Symmetry 14.11 (2022): 2304. [2] Kim, Song-Kyoo, et al. \"Advanced machine learning based malware detection systems.\" IEEE Access 12 (2024): 115296–115305. [3] Chowdhury, Naseef-Ur-Rahman, et al. \"Android malware detection using machine learning: A review.\" Intelligent Systems Conference. Cham: Springer Nature Switzerland, 2023. [4] Gaber, Matthew G., Mohiuddin Ahmed, and Helge Janicke. \"Malware detection with artificial intelligence: A systematic literature review.\" ACM Computing Surveys 56.6 (2024): 1–33. [5] Brown, Austin, Maanak Gupta, and Mahmoud Abdelsalam. \"Automated machine learning for deep learning based malware detection.\" Computers & Security 137 (2024): 103582. [6] Alraizza, Amjad, and Abdulmohsen Algarni. \"Ransomware detection using machine learning: A survey.\" Big Data and Cognitive Computing 7.3 (2023): 143. [7] Alomari, Esraa Saleh, et al. \"Malware detection using deep learning and correlation-based feature selection.\" Symmetry 15.1 (2023): 123. [8] Gorment, Nor Zakiah, et al. \"Machine learning algorithm for malware detection: Taxonomy, current challenges, and future directions.\" IEEE Access 11 (2023): 141045–141089. [9] Herrera-Silva, Juan A., and Myriam Hernandez-Alvarez. \"Dynamic feature dataset for ransomware detection using machine learning algorithms.\" Sensors 23.3 (2023): 1053. [10] Wu, Yinwei, et al. \"DroidRL: Feature selection for Android malware detection with reinforcement learning.\" Computers & Security 128 (2023): 103126.

Copyright

Copyright © 2026 Harish B, Sankara Narayanan S T. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.